fix(PythonAPI/build): add NumPy 2 compatibility via Boost 1.90 upgrade - #9670
Conversation
|
Why should we merge this? |
|
Fair enough. I expect NumPy 1.x to reach end of life, and downstream libraries are already moving to require NumPy 2 (e.g., opencv-python >= 4.12). Having this compatibility in place before it becomes urgent seems better than scrambling later, but I understand if the priority isn't there yet. |
|
What about using updated version of boost instead of patching it? I do not like patches. |
|
Good point. I went with a patch to avoid a full Boost version bump, which could be quite invasive in this refactoring, since we only needed to change a few things from Boost. But if you’d prefer to upgrade Boost instead, I can look into it and send a new commit. |
|
is a PR from a former CARLA collaborator. |
8aa6489 to
b3f4d55
Compare
JArmandoAnaya
changed the title
JArmandoAnaya
changed the title
… compatibility Replace np.bool with np.bool_ in DVS event dtype in V2XDemo.py, manual_control_chrono.py, and manual_control_carsim.py (removed in NumPy 1.24). Replace np.matrix with np.array in client_bounding_boxes.py (deprecated in NumPy 1.x, removed in NumPy 2.x). All replacements are backward-compatible with NumPy >= 1.18.4.
Drop the in-tree dtype.cpp.patch against Boost 1.84.0 and upgrade to Boost 1.89.0, which carries the upstream NumPy 2 C ABI fix from boostorg/python#432 natively. libboost_numpy now compiles against both NumPy 1.x (>=1.18.4) and NumPy 2.x headers without any vendored patches. Boost 1.84 -> 1.89 removed several deprecated Asio and Filesystem APIs; all call sites in LibCarla and the CarlaTools UE4 plugin updated: - io_context::work -> executor_work_guard (removed 1.85) - io_context::reset() -> restart() (removed 1.86) - address::from_string() -> make_address() (removed 1.85) - resolver::query/iterator -> range-based resolve() (removed 1.85) - buffer_cast<T>() -> static_cast via streambuf::data().data() (removed 1.88) - io_service typedef -> io_context (removed 1.85) - boost/filesystem/operations.hpp no longer includes directory.hpp (1.85) Windows installer (install_boost.bat) SHA256 updated to the 1.89.0 zip digest. MSVC C4459 pragma guard added around boost/asio.hpp in MapPreviewUserWidget.h to prevent warnings-as-errors failure. New regression-guard tests: test_boost_deadline_timer.cpp (3 tests), test_boost_rtree.cpp (4 tests), test_boost_version.py.
Boost 1.90 added [[deprecated]] annotations to boost::asio::deadline_timer.
LibCarla's networking code is migrated to boost::asio::steady_timer across
Primary, Secondary, Client, and ServerSession.
steady_timer default-constructs with expiry() == time_point::min() (past),
whereas deadline_timer defaulted to pos_infin (far future). The recursive
StartTimer() guard in ServerSession::StartTimer and Primary::StartTimer
checks expiry() <= now() and would fire Close() before socket setup if left
at the new default, so both _deadline members are explicitly constructed
with time_point::max() to preserve the old semantics.
API migrations at the call sites:
* expires_from_now(duration) -> expires_after(duration)
* expires_at() -> expiry()
* deadline_timer::traits_type::now() -> std::chrono::steady_clock::now()
* boost::posix_time durations -> std::chrono durations
Build scripts:
* Util/BuildTools/Setup.sh: BOOST_VERSION 1.89.0 -> 1.90.0, SHA256
* Util/BuildTools/Setup.bat: version bump (Windows)
* Util/BuildTools/BuildOSMRenderer.bat: install path bump (Windows)
* Util/InstallersWin/install_boost.bat: source-zip SHA256 bump
Tests:
* test_boost_deadline_timer.cpp: test class BoostDeadlineTimer renamed
to BoostAsioTimer and migrated to steady_timer; 3 new regression
tests pin the contract that prevents future refactors from silently
reintroducing the default-construction trap:
- InitializedWithMaxIsInDistantFuture
- DefaultConstructedExpiryIsInPast
- CancelTriggersAbortedError
* test_boost_rtree.cpp: header comment 1.89 -> 1.90
* test_boost_version.py: EXPECTED_BOOST_VERSION 1.89.0 -> 1.90.0;
stale-cache check rolled 1.84.0 -> 1.89.0
* CHANGELOG.md: NumPy 2 entry references Boost 1.90.0
Verified: make LibCarla + full make check.LibCarla pass on server and
client, release and debug. All 6 BoostAsioTimer.* and 4 BoostRtree.* tests
pass on both suites.
7ff59bc to
5e68bef
Compare
|
Ready! I upgraded Boost to 1.90. I used the example from @MarcelPiNacy-CVC as the first try. But, after that, migrating from 1.89 to 1.90 required only a few extra steps, so I did it in a second commit. |
There was a problem hiding this comment.
Great work on this PR!
The API migrations are all correct, the time_point::max() initialization fix is a subtle but important catch, and the test coverage is thorough.
Thanks for the effort put into this!
One small observation: test_no_boost_189_in_linked_libs in test_boost_version.py guards against accidentally linking a stale boost-1.89.0 install (the intermediate version used during development of this PR), but a user upgrading directly from the previous 1.84.0 baseline would slip past it undetected.
Since that's arguably the more common stale-cache scenario for most contributors, it might be worth adding a similar check for boost-1.84.0 there too or widening the assertion to something like "must link against boost-1.90.0" rather than "must not link against boost-1.89.0"
Address review on PR carla-simulator#9670. The previous test guarded only against the 1.89.0 intermediate install used during development, letting a direct 1.84.0 -> 1.90.0 upgrade slips past undetected. - Replace the negative-only LDD check with a positive assertion that libboost_*.so.1.90.0 is present in PythonAPI/carla/dependencies/lib/, plus an explicit guard over known stale baselines (1.84.0, 1.89.0). Boost is statically linked into libcarla.so, so ldd cannot observe it; the dependencies/lib symlinks are the authoritative record. - Update two Linux build scripts still pinned to boost-1.84.0 that were missed in the 1.84->1.89->1.90 bumps (the Windows .bat was updated): - Util/BuildTools/BuildOSMRenderer.sh - Util/BuildTools/Ad-rss.sh
|
Hello! Thanks for the code review. I made the update. |
3 participants
Description
Upgrades Boost from 1.84.0 to 1.90.0 and, as a direct consequence, makes CARLA's
PythonAPI compatible with both NumPy 1.x (>= 1.18.4) and NumPy 2.x. NumPy 1.x
support is preserved, not dropped.
We first landed the upgrade at Boost 1.89.0, which is the minimum version that
ships the upstream NumPy 2 C ABI fix
(boostorg/python#432) natively.
Moving from 1.89.0 to 1.90.0 was a small follow-up: 1.90 only added
[[deprecated]]annotations toboost::asio::deadline_timer, which we migratedto
boost::asio::steady_timer. Going straight to the newest stable releaseavoids carrying a known-deprecated API into the tree.
Improvements
boost-1.84.0-install/trees are picked up automatically via the updatedBOOST_VERSIONand SHA256 pins inUtil/BuildTools/Setup.sh,Util/BuildTools/Setup.bat,Util/BuildTools/BuildOSMRenderer.bat, andUtil/InstallersWin/install_boost.bat.boost::asio::deadline_timertoboost::asio::steady_timerinPrimary,Secondary,Client, andServerSession. No deprecated Asio API remains in the tree.numpy<2.0.0tonumpy>=1.18.4inPythonAPI/examples/requirements.txt,PythonAPI/carla/agents/requirements.txt,and
PythonAPI/test/requirements.txt.Util/Docker/requirements/3.11/test.txtand3.12/test.txt) pinned tonumpy>=2.0.0to exercise NumPy 2.x on CI.Util/Patches/boost-1.84.0/dtype.cpp.patchand its helpers(
verify_patch.cpp,verify_patch.sh) are deleted. Boost ships the fix now.Fixes
NumPy 2 Python-level removals.
np.boolreplaced withnp.bool_inPythonAPI/examples/V2XDemo.py,manual_control_chrono.py, andmanual_control_carsim.py(DVS event dtype).np.matrix(np.identity(4))replaced with
np.array(np.identity(4))inclient_bounding_boxes.py, safebecause the only consumer is
np.dot().NumPy 2 C ABI (NEP 52).
Boost 1.89+ makes
libboost_numpybuild against both NumPy 1.x and 2.xheaders without any vendored patches (
PyArray_Descris now opaque).Boost 1.84 -> 1.90 API call-site updates in LibCarla and the CarlaTools
UE4 plugin (each removal was introduced between 1.85 and 1.90):
LibCarla/source/carla/ThreadPool.hio_context::work->executor_work_guardLibCarla/source/carla/multigpu/listener.cppio_context::reset()->restart()LibCarla/source/carla/multigpu/router.cpp,secondary.cppaddress::from_string()->make_address()LibCarla/source/carla/rpc/Server.hreset()->restart()LibCarla/source/carla/streaming/EndPoint.hresolver::query/iterator-> range-basedresolve()LibCarla/source/carla/FileSystem.cpp#include <boost/filesystem/directory.hpp>(split in 1.85)Unreal/.../MapPreviewUserWidget.{h,cpp}io_service->io_contextbuffer_cast<T>()->static_castviastreambuf::data().data()MSVC C4459 pragma guard
steady_timerdefault-construction trap.deadline_timerdefaulted topos_infin, butsteady_timerdefault-constructs withexpiry() == time_point::min()(in the past).
Primary::StartTimerandServerSession::StartTimerguard onexpiry() <= now()to detect timeouts, so the default would fireClose()before the socket is set up.
_deadlineis now explicitly initialised withstd::chrono::steady_clock::time_point::max()in both constructors.Call-site API migrations for the timer rewrite:
expires_from_now(d)->expires_after(d),expires_at()->expiry(),deadline_timer::traits_type::now()->std::chrono::steady_clock::now(),boost::posix_timedurations ->std::chronodurations.Added testing
New regression guards added under
LibCarla/source/test/common/andPythonAPI/test/unit/:test_boost_deadline_timer.cpp(test class
BoostAsioTimer)TimerFiresWithinReasonableTime,ExpiresAtNowComparison,ChronoDurationComposition,InitializedWithMaxIsInDistantFuture(pins thetime_point::max()fix),DefaultConstructedExpiryIsInPast(documents the trap),CancelTriggersAbortedError(covers theCloseNow()path)test_boost_rtree.cppgeometry::index::rtree(used by TrafficManager)test_boost_version.pySetup.shis current (1.90.0) and no staleboost-1.84.0patch dir or link existstest_numpy_compat.pyrun_numpy_compat_matrix.shtest_numpy_compat.pyin eachCommits
d4151ed05fix(PythonAPI/examples): replace deprecated NumPy aliases69df4d2a1fix(build): upgrade Boost to 1.89.0 for NumPy 2 ABI compatibilitya5e4ce0d3docs(CHANGELOG): note NumPy 2 compatibility5e68bef2cfix(build): bump Boost 1.89 -> 1.90 and migrate timers to steady_timerFixes #9365
Where has this been tested?
Results:
make check.LibCarlapasses in both debug and release on the server andclient suites, including all 6
BoostAsioTimer.*and 4BoostRtree.*tests.make check.PythonAPIpasses, including the fulltest_boost_version.pyset (static Boost version pin + no stale 1.84.0 link).
python3 -m nose2 -v --start-dir PythonAPI/test/unit test_numpy_compatonPython 3.12 + NumPy 2.4.4 with the carla wheel installed: 15/15 pass.
bash PythonAPI/test/unit/run_numpy_compat_matrix.sh python3.12:NumPy 1.26.4 venv 15/15 pass, NumPy 2.4.4 venv 15/15 pass. The 2.x log
confirms end-to-end load:
carla C extension OK with NumPy 2.4.4, NumPy 2.x (NEP 52, opaque PyArray_Descr).make package ARGS="--python-version=3.10,3.11,3.12 --ros2 --chrono --no-zip --target-wheel-platform=manylinux_2_31_x86_64": SUCCESS.Possible Drawbacks
Build/boost-1.84.0-c10-install/orBuild/boost-1.84.0-install/tree from a previous checkout need to remove itbefore building, so the 1.90.0 source is fetched and compiled.
make cleanis sufficient.
install_boost.batSHA256 has been updated to the 1.90.0 source-zipdigest. Cached
boost-1.84.0-install/directories on Windows build agentsmust likewise be removed. The Windows CI path does not build ROS2, so none of
the Asio/Filesystem call-site changes affect the Windows CI build.
This change is